Transition from E+ 23.2.0 to E+ 24.1.0

---

Foreword

This notebook has two big parts:

• Part 1: aims to transition all regression tests from one E+ version to the next, and allows you to run each test in both the old and the new version
  • These will take quite some time to run (about 1hr to run the tests in the OLD OpenStudio version, transition the IDFs to the new E+ version and run them, and run the tests in the NEW OpenStudio Version, based on almost 200 files currently)
  • By default it will just copy over the SQL from the regression test to place in OLD_DIR, but if you want to force rerun the IDF in the old E+ version you can.
  • At the end of Part 1, you will have three CSV files, one per version, with the site KBTUs for each test. And you also have an organized tree of VERSION/TEST_NAME/ output directories that have the SQL files we will use for sections 6+.
• Part 2: aims to analyze the differences between versions
  • Section 3.1 just re-queries all SQL file (or you can reload the three CSV files) to highlight the tests with the biggest site KBTU differences
  • Section 3.2 provides a high-level interface that only requires to pass a test name and it will query the relevant SQL files and produce visualization (tables, grouped bar charts, and heatmaps) to analyze where differences may be coming from
  If you have already run Part 1 successfully once, you only need to run Section 1. and you can jump to Part 2 directly.

PART 1: Transition all regression tests and run them in both E+ versions

Part 1 is ommited in the HTML because the interesting part is the output and analysis at the end

PART 2: Analyzing differences

Analyzing E+ runtime

THIS NO LONGER WORKS, last version where it worked was 9.2.0... I don't know where the runtime went in the SQLFile

Concat frames and look at files that failed to run

These are the files were we have some (but not all) failures

E+	23.2.0-7636e6b3e9	24.1.0-9d7789a3ac
OS	3.7.0	Transition	3.8.0
python_plugin.osm	1044617.682811	nan	nan
python_plugin.rb	1044617.682811	1044617.682811	nan
storage_liion_battery.osm	2089235.365622	nan	2089235.365622
storage_liion_battery.rb	2089235.365622	nan	2089235.365622

Rerun transition files that need expand objects (can't do it in parallel)

First look at where we have deviations

Setting % diff threshold to 0.010%

Deviations in Transition and/or new OpenStudio

Deviations from Transition to new OS only

Deviations from Transition to new OS only: Differences above 0.010%

E+	24.1.0-9d7789a3ac
OS	Transition	3.8.0
vrf_fluid_temperature_control.rb	0.004%	-0.011%
vrf_fluid_temperature_control.osm	0.004%	-0.011%
multi_stage_electric.rb	-0.073%	0.165%
multi_stage.rb	-0.166%	-0.014%
multi_stage_electric.osm	-0.270%	0.225%
centralheatpumpsystem.osm	-0.557%	-0.011%
centralheatpumpsystem.rb	-0.777%	-0.010%

Grouped bar chart of differences compared to the old OpenStudio

How to read this chart :

The percentage differences are calculated compared to the Old OpenStudio results for both the transitioned results and the new OpenStudio results.

What you need to pay special attention to is when you don't have the same difference between the Transition to Old OS and the New OS to old OS (meaning the difference is not E+'s fault, but OpenStudio's).

OS	Transition	3.8.0
centralheatpumpsystem.rb	-0.78%	-0.79%
centralheatpumpsystem.osm	-0.56%	-0.57%
heatpump_airtowater_fuelfired.osm	0.37%	0.37%
heatpump_airtowater_fuelfired.rb	0.37%	0.37%
lowtemprad_constflow.osm	0.35%	0.35%
daylighting_no_shades.osm	0.30%	0.30%
daylighting_shades.osm	0.29%	0.29%
photovoltaics.osm	0.29%	0.29%
pv_and_storage_facilityexcess.osm	0.29%	0.29%
transformer.osm	0.29%	0.29%
unitary_systems_airloop_and_zonehvac.osm	0.29%	0.29%
plantloop_avms.osm	0.29%	0.29%
epw_design_conditions.osm	0.29%	0.29%
additional_props.osm	0.29%	0.29%
pv_and_storage_demandleveling.osm	0.29%	0.29%
moisture_settings.osm	0.29%	0.29%
utility_bill02.osm	0.29%	0.29%
utility_bill01.osm	0.28%	0.28%
interior_partitions.osm	0.27%	0.27%
solar_collector_flat_plate_water.osm	0.27%	0.27%
multi_stage_electric.osm	-0.27%	-0.05%
adiabatic_construction_set.rb	0.27%	0.27%
solar_collector_flat_plate_photovoltaicthermal.osm	0.26%	0.26%
solar_collector_integralcollectorstorage.osm	0.26%	0.26%
water_heaters.osm	0.26%	0.26%
swimmingpool_indoor.osm	-0.24%	-0.24%
swimmingpool_indoor.rb	-0.24%	-0.24%
surface_properties_ground_and_solarmult.rb	0.19%	0.19%
surface_properties_ground_and_solarmult.osm	0.19%	0.19%
surface_properties_lwr.rb	0.18%	0.18%
surface_properties_lwr.osm	0.18%	0.18%
multi_stage.rb	-0.17%	-0.18%
adiabatic_construction_set.osm	0.18%	0.18%
outputcontrol_files.osm	0.18%	0.18%
outputcontrol_timestamp.rb	0.18%	0.18%
outputcontrol_files.rb	0.18%	0.18%
output_objects_2.osm	0.18%	0.18%
outputcontrol_timestamp.osm	0.18%	0.18%
output_objects_2.rb	0.18%	0.18%
photovoltaics_sandia.rb	0.18%	0.18%
photovoltaics_sandia.osm	0.18%	0.18%
output_objects.osm	0.17%	0.17%
output_objects.rb	0.17%	0.17%
window_property_frame_and_divider.osm	0.17%	0.17%
window_property_frame_and_divider.rb	0.17%	0.17%
foundation_kiva_customblocks.osm	0.17%	0.17%
foundation_kiva.osm	0.17%	0.17%
unitary_system.rb	0.16%	0.16%
unitary_system.osm	0.16%	0.16%
pthp_othercoils.osm	0.15%	0.15%
zone_hvac.rb	0.15%	0.15%
zone_hvac_equipment_list.osm	0.15%	0.15%
zone_hvac_equipment_list.rb	0.15%	0.15%
zone_property_user_view_factors_by_surface_name.osm	0.14%	0.14%
unitary_vav_bypass_coiltypes.rb	0.13%	0.14%
unitary_test.osm	0.13%	0.13%
unitary_test.rb	0.13%	0.13%
unitary_vav_bypass_coiltypes.osm	0.13%	0.13%
coil_waterheating_desuperheater.osm	0.13%	0.13%
multi_stage.osm	-0.12%	-0.11%
coilsystem_dxhx.rb	0.12%	0.12%
coilsystem_dxhx_desiccant_balancedflow.osm	0.12%	0.12%
coilsystem_dxhx_desiccant_balancedflow.rb	0.12%	0.12%
coilsystem_dxhx.osm	0.12%	0.12%
generator_microturbine.osm	0.12%	0.12%
heatexchanger_desiccant_balancedflow.rb	0.12%	0.12%
heatexchanger_desiccant_balancedflow.osm	0.12%	0.12%
roof_vegetation.osm	0.11%	0.11%
roof_vegetation.rb	0.11%	0.11%
exterior_equipment.osm	0.11%	0.11%
exterior_equipment.rb	0.11%	0.11%
space_load_instances.osm	0.10%	0.10%
space_load_instances.rb	0.10%	0.10%
heatpump_varspeed.osm	0.10%	0.10%
heatpump_varspeed.rb	0.10%	0.10%
dsn_oa_w_ideal_loads.rb	0.10%	0.10%
shadingcontrol_singlezone.osm	0.10%	0.10%
shadingcontrol_singlezone.rb	0.10%	0.10%
daylighting_devices.rb	0.10%	0.10%
ExampleModel.rb	-0.09%	-0.09%
lifecyclecostparameters.rb	-0.09%	-0.09%
multi_stage_electric.rb	-0.07%	0.09%
unitary_vav_bypass_plenum.osm	0.09%	0.09%
baseline_sys02.osm	0.08%	0.08%
chillers_tertiary.osm	0.08%	0.08%
chillers_tertiary.rb	0.08%	0.08%
dsn_oa_w_ideal_loads.osm	0.08%	0.08%
humidity_control.rb	0.08%	0.08%
water_economizer.rb	0.07%	0.07%
unitary_vav_bypass.osm	0.07%	0.07%
unitary_vav_bypass.rb	0.07%	0.07%
zone_control_contaminant_controller.rb	0.07%	0.07%
heatpump_plantloop_eir.osm	0.07%	0.07%
heatpump_plantloop_eir.rb	0.07%	0.07%
zone_control_contaminant_controller.osm	0.07%	0.07%
humidity_control.osm	0.07%	0.07%
surfacecontrol_moveableinsulation.osm	0.07%	0.07%
surfacecontrol_moveableinsulation.rb	0.07%	0.07%
coil_cooling_dx.rb	0.07%	0.07%
coil_cooling_dx.osm	0.07%	0.07%
airloop_and_zonehvac.osm	0.07%	0.07%
generator_windturbine.rb	0.07%	0.07%
generator_windturbine.osm	0.07%	0.07%
sizing_zone_dszad.osm	0.07%	0.07%
sizing_zone_dszad.rb	0.07%	0.07%
shadowcalculation.osm	0.07%	0.07%
shadowcalculation.rb	0.07%	0.07%
coil_waterheating_desuperheater.rb	0.07%	0.07%
daylighting_devices.osm	0.07%	0.07%
coil_userdefined.rb	-0.07%	-0.07%
coil_userdefined.osm	-0.07%	-0.07%
zone_hvac2.rb	0.06%	0.07%
chiller_electric_ashrae205.rb	0.07%	0.07%
chiller_electric_ashrae205.osm	0.07%	0.07%
baseline_sys01.osm	0.07%	0.07%
pvwatts.osm	0.07%	0.07%
baseline_sys03.rb	0.07%	0.07%
hot_water.rb	0.07%	0.07%
evaporative_cooling.rb	0.07%	0.07%
ideal_plant.rb	0.07%	0.07%
ideal_plant.osm	0.07%	0.07%
hot_water.osm	0.07%	0.07%
air_chillers.osm	0.06%	0.06%
zone_hvac2.osm	0.06%	0.06%
availability_managers.osm	-0.06%	-0.06%
baseline_sys06.rb	0.06%	0.06%
fuelcell.rb	0.06%	0.06%
fuelcell.osm	0.06%	0.06%
surface_properties.osm	0.06%	0.06%
pv_and_storage_demandleveling.rb	0.06%	0.06%
baseline_sys08.rb	0.06%	0.06%
utility_bill01.rb	0.06%	0.06%
vrf_watercooled.osm	0.06%	0.06%
daylighting_no_shades.rb	0.06%	0.06%
ems.rb	0.06%	0.06%
epw_design_conditions.rb	0.06%	0.06%
additional_props.rb	0.06%	0.06%
plantloop_avms.rb	0.06%	0.06%
baseline_sys01.rb	0.06%	0.06%
pvwatts.rb	0.06%	0.06%
transformer.rb	0.06%	0.06%
photovoltaics.rb	0.06%	0.06%
utility_bill02.rb	0.06%	0.06%
daylighting_shades.rb	0.06%	0.06%
pv_and_storage_facilityexcess.rb	0.06%	0.06%
solar_collector_flat_plate_photovoltaicthermal.rb	0.06%	0.06%
moisture_settings.rb	0.06%	0.06%
plenums.osm	0.06%	0.06%
plenums.rb	0.06%	0.06%
solar_collector_flat_plate_water.rb	0.05%	0.05%
water_heaters.rb	0.05%	0.05%
airterminal_cooledbeam.osm	0.05%	0.05%
solar_collector_integralcollectorstorage.rb	0.05%	0.05%
availability_managers.rb	-0.05%	-0.05%
vrf_airloophvac.osm	0.04%	0.04%
vrf_watercooled.rb	0.04%	0.04%
foundation_kiva_customblocks.rb	0.04%	0.04%
vrf_airloophvac.rb	0.04%	0.04%
baseline_sys09.rb	0.04%	0.04%
dist_ht_cl.rb	0.04%	0.04%
setpoint_manager_systemnodereset.osm	0.04%	0.04%
setpoint_manager_systemnodereset.rb	0.04%	0.04%
foundation_kiva.rb	0.04%	0.04%
headered_pumps.rb	0.04%	0.04%
baseline_sys04.rb	0.04%	0.04%
baseline_sys02.rb	0.04%	0.04%
heatexchanger_airtoair_sensibleandlatent.rb	0.04%	0.04%
solar_collector_flat_plate_photovoltaicthermal_bipvt.osm	0.04%	0.04%
solar_collector_flat_plate_photovoltaicthermal_bipvt.rb	0.04%	0.04%
refrigeration_system_2.rb	0.04%	0.04%
refrigeration_system_2.osm	0.04%	0.04%
baseline_sys10.rb	0.04%	0.04%
airloop_avms.osm	0.04%	0.04%
airloop_avms.rb	0.04%	0.04%
ptac_othercoils.osm	0.04%	0.04%
zone_fan_exhaust.rb	0.04%	0.04%
fan_on_off.rb	0.04%	0.04%
refrigeration_system.osm	0.04%	0.04%
refrigeration_system.rb	0.04%	0.04%
fan_componentmodel.rb	0.04%	0.04%
fan_componentmodel.osm	0.04%	0.04%
air_terminals.rb	0.04%	0.04%
schedule_file.rb	0.04%	0.04%
schedule_file.osm	0.03%	0.03%
unitary_vav_bypass_plenum.rb	-0.03%	-0.03%
surface_properties.rb	0.03%	0.03%
baseline_sys06.osm	0.03%	0.03%
pthp_othercoils.rb	0.03%	0.03%
vrf.rb	0.03%	0.03%
baseline_sys08.osm	0.03%	0.03%
zone_property_user_view_factors_by_surface_name.rb	0.03%	0.03%
baseline_sys05.rb	0.03%	0.03%
plant_op_deltatemp_schemes.rb	0.03%	0.03%
plant_op_deltatemp_schemes.osm	0.03%	0.03%
cooling_coils.osm	0.03%	0.03%
cooling_coils.rb	0.03%	0.03%
baseline_sys04.osm	0.03%	0.03%
baseline_sys07.rb	0.03%	0.03%
thermal_storage.osm	0.03%	0.03%
meters.osm	0.03%	0.03%
meters.rb	0.03%	0.03%
thermal_storage.rb	0.03%	0.03%
plant_op_schemes.osm	0.03%	0.03%
plant_op_schemes.rb	0.03%	0.03%
lowtemprad_constflow.rb	-0.03%	-0.03%
plant_op_temp_schemes.rb	0.03%	0.03%
plant_op_temp_schemes.osm	0.03%	0.03%
fan_systemmodel.osm	0.03%	0.03%
absorption_chillers.rb	0.03%	0.03%
absorption_chillers.osm	0.03%	0.03%
fan_systemmodel.rb	0.03%	0.03%
unitary_systems_airloop_and_zonehvac.rb	0.03%	0.03%
ideal_loads_w_plenums.rb	0.03%	0.03%
ideal_loads_w_plenums.osm	0.03%	0.03%
zone_hvac.osm	-0.03%	-0.03%
generator_microturbine.rb	0.03%	0.03%
ducts_and_pipes.osm	-0.02%	-0.02%
baseline_sys03.osm	0.02%	0.02%
dist_ht_cl.osm	0.02%	0.02%
setpoint_managers.osm	0.02%	0.02%
setpoint_managers.rb	0.02%	0.02%
headered_pumps.osm	0.02%	0.02%
ducts_and_pipes.rb	-0.02%	-0.02%
tablelookup.rb	0.02%	0.02%
tablelookup.osm	0.02%	0.02%
vrf.osm	0.02%	0.02%
airloop_and_zonehvac.rb	0.02%	0.02%
ptac_othercoils.rb	0.02%	0.02%
coil_waterheating_desuperheater_2.rb	0.02%	0.02%
coil_waterheating_desuperheater_2.osm	0.02%	0.02%
heatpump_hot_water.osm	0.02%	0.02%
heatpump_hot_water.rb	0.02%	0.02%
zone_fan_exhaust.osm	0.02%	0.02%
coolingtowers.rb	0.02%	0.02%
performanceprecisiontradeoffs.osm	0.02%	0.02%
performanceprecisiontradeoffs.rb	0.02%	0.02%
airterminal_cooledbeam.rb	0.02%	0.02%
interior_partitions.rb	0.02%	0.02%
vrf_fluid_temperature_control_hr.rb	0.02%	0.02%
vrf_fluid_temperature_control_hr.osm	0.02%	0.02%
fan_on_off.osm	0.02%	0.02%
fluid_coolers.rb	-0.02%	-0.02%
fluid_coolers.osm	-0.02%	-0.02%
dual_duct.osm	0.02%	0.02%
dual_duct.rb	0.02%	0.02%
heatexchanger_airtoair_sensibleandlatent.osm	0.02%	0.02%
baseline_sys09.osm	0.02%	0.02%
baseline_sys10.osm	0.01%	0.01%
coilsystem_waterhx.rb	0.01%	0.01%
coilsystem_waterhx.osm	0.01%	0.01%
hightemprad.osm	-0.01%	-0.01%
hightemprad.rb	-0.01%	-0.01%
heatrecovery_chiller.rb	-0.01%	-0.01%
heatrecovery_chiller.osm	-0.01%	-0.01%
schedule_ruleset_2012_NonLeapYear.rb	0.01%	0.01%
schedule_ruleset_2012_LeapYear.rb	0.01%	0.01%
schedule_ruleset_2012_LeapYear.osm	0.01%	0.01%
schedule_ruleset_2012_NonLeapYear.osm	0.01%	0.01%
humidity_control_2.rb	0.01%	0.01%
humidity_control_2.osm	0.01%	0.01%
schedule_ruleset_2013.rb	0.01%	0.01%
schedule_fixed_interval.rb	0.01%	0.01%
schedule_fixed_interval_schedulefile.rb	0.01%	0.01%
schedule_ruleset_2013.osm	0.01%	0.01%

Stacked bar chart of differences

How to read this chart :

The percentage differences are calculated compared to the Old OpenStudio results for both the transitioned results and the new OpenStudio results. Then I do

% diff New version = % diff New version - % diff Transition

and plot that as a stacked bar chart. The goal is to more clearly see the differences that are due to OpenStudio by removing the differences due to the new E+.

What you need to pay special attention to is when you see % differences for the new OS.

A cursor along with the total % difference between Old OS and New OS is also plotted . Please see the below example to get a better sense of how the graph is constructed

Let's do the actual plotting now: